Skip to content

feat(go/ai): added DefineMiddleware (Middleware V2)#4464

Open
apascal07 wants to merge 24 commits intomainfrom
ap/go-middleware
Open

feat(go/ai): added DefineMiddleware (Middleware V2)#4464
apascal07 wants to merge 24 commits intomainfrom
ap/go-middleware

Conversation

@apascal07
Copy link
Copy Markdown
Collaborator

@apascal07 apascal07 commented Feb 6, 2026

Adds a Middleware interface for wrapping generation, model calls, and tool execution with composable hooks, plus the registration and plugin plumbing needed to expose middleware to the Dev UI and cross-runtime tooling. Built-in implementations (retry, fallback, tool approval, filesystem, skills) ship in a follow-up PR.

The core mental model: a middleware is a config struct with two methods. Name() returns its stable registered identifier and New(ctx) produces a per-call bundle of hook functions.

Examples

Attaching middleware to a Generate call

Middleware is attached per call with ai.WithUse. The chain applies outer-to-inner, so ai.WithUse(A, B) expands to A { B { model } }:

response, _ := genkit.Generate(ctx, g,
    ai.WithModelName("googleai/gemini-flash-latest"),
    ai.WithPrompt("Explain quantum computing."),
    ai.WithUse(
        Logger{Prefix: "[trace]"},
        SomeOtherMiddleware{},
    ),
)

No plugin registration is required for pure-Go use. WithUse calls the config's New directly on the local fast path. Registration is what makes middleware visible to the Dev UI and callable from other runtimes.

Defining your own middleware

Implement ai.Middleware by providing two methods on a config struct. Name() returns the registered identifier (a stable constant read from the zero value of the type). New(ctx) returns an *ai.Hooks value with any of WrapGenerate, WrapModel, WrapTool, and Tools populated. Nil fields pass through.

type Logger struct {
    Prefix string `json:"prefix,omitempty"`
}

func (Logger) Name() string { return "mine/logger" }

func (l Logger) New(ctx context.Context) (*ai.Hooks, error) {
    return &ai.Hooks{
        WrapModel: func(ctx context.Context, p *ai.ModelParams, next ai.ModelNext) (*ai.ModelResponse, error) {
            start := time.Now()
            resp, err := next(ctx, p)
            log.Printf("%s model call took %s", l.Prefix, time.Since(start))
            return resp, err
        },
    }, nil
}

Per-call state is closure-captured inside New (which runs once per Generate invocation). Plugin-level state belongs on unexported fields of the config struct; the plugin's Middlewares() method sets those fields on a prototype that value-copies across JSON-dispatched calls to preserve them.

Ad-hoc inline middleware

For one-off middleware that does not need Dev UI visibility or a named type, use ai.MiddlewareFunc:

ai.WithUse(ai.MiddlewareFunc(func(ctx context.Context) (*ai.Hooks, error) {
    return &ai.Hooks{
        WrapModel: func(ctx context.Context, p *ai.ModelParams, next ai.ModelNext) (*ai.ModelResponse, error) {
            log.Printf("model call")
            return next(ctx, p)
        },
    }, nil
}))

Plugin-provided middleware

Plugins advertise middleware by implementing MiddlewarePlugin. Middlewares is called during genkit.Init, and the returned descriptors become visible to the Dev UI and available for lookup by name. Any unexported fields on the prototype (like client below) are preserved across Dev UI and cross-runtime invocations via value-copy in the descriptor's build closure.

type MyPlugin struct{
    client *http.Client
}

func (p *MyPlugin) Name() string                          { return "myplugin" }
func (p *MyPlugin) Init(ctx context.Context) []api.Action { return nil }

func (p *MyPlugin) Middlewares(ctx context.Context) ([]*ai.MiddlewareDesc, error) {
    return []*ai.MiddlewareDesc{
        ai.NewMiddleware("description of my middleware", Logger{client: p.client}),
    }, nil
}

API Reference

ai.Middleware interface

// A middleware is identified by a stable name and produces a fresh hooks
// bundle for each Generate() call.
type Middleware interface {
    Name() string
    New(ctx context.Context) (*Hooks, error)
}

ai.Hooks bundle

type Hooks struct {
    Tools        []Tool
    WrapGenerate func(ctx context.Context, params *GenerateParams, next GenerateNext) (*ModelResponse, error)
    WrapModel    func(ctx context.Context, params *ModelParams, next ModelNext) (*ModelResponse, error)
    WrapTool     func(ctx context.Context, params *ToolParams, next ToolNext) (*MultipartToolResponse, error)
}
  • WrapGenerate wraps each iteration of the tool loop (invoked N+1 times for a generate with N tool turns).
  • WrapModel wraps each model API call.
  • WrapTool wraps each tool execution and may run concurrently for parallel tool calls.
  • Tools contributes additional tools to register for the generation.

Hook parameters

// GenerateParams holds params for the WrapGenerate hook.
type GenerateParams struct {
    Options      *GenerateActionOptions // original options passed to Generate
    Request      *ModelRequest          // current model request with accumulated messages
    Iteration    int                    // zero-indexed tool-loop iteration
    MessageIndex int                    // next index for streamed response chunks
    Callback     ModelStreamCallback    // streaming callback, nil if not streaming
}

// ModelParams holds params for the WrapModel hook.
type ModelParams struct {
    Request  *ModelRequest
    Callback ModelStreamCallback
}

// ToolParams holds params for the WrapTool hook.
type ToolParams struct {
    Request *ToolRequest
    Tool    Tool
}

// Chain continuation functions invoked by hooks to pass control to the next layer.
type GenerateNext = func(ctx context.Context, params *GenerateParams) (*ModelResponse, error)
type ModelNext    = func(ctx context.Context, params *ModelParams) (*ModelResponse, error)
type ToolNext     = func(ctx context.Context, params *ToolParams) (*MultipartToolResponse, error)

Registration

// NewMiddleware constructs a descriptor without registering it. The prototype
// supplies both the registered name (via its Name() method) and any plugin-
// level state that should flow into JSON-dispatched invocations via unexported
// fields preserved by value-copy.
func NewMiddleware[M Middleware](description string, prototype M) *MiddlewareDesc

// DefineMiddleware creates and registers a middleware descriptor.
func DefineMiddleware[M Middleware](r api.Registry, description string, prototype M) *MiddlewareDesc

// LookupMiddleware returns the registered middleware descriptor with the given name.
func LookupMiddleware(r api.Registry, name string) *MiddlewareDesc

Matching conveniences exist in the genkit package:

func genkit.DefineMiddleware[M ai.Middleware](g *Genkit, description string, prototype M) *ai.MiddlewareDesc
func genkit.LookupMiddleware(g *Genkit, name string) *ai.MiddlewareDesc

Application code typically uses genkit.DefineMiddleware to register middleware it owns directly. Plugin authors typically use ai.NewMiddleware and return descriptors from MiddlewarePlugin.Middlewares(); genkit.Init registers them during plugin setup.

Inline adapter

// MiddlewareFunc adapts a per-call factory closure to the Middleware interface
// for ad-hoc inline middleware, without a registered descriptor or plugin wiring.
type MiddlewareFunc func(ctx context.Context) (*Hooks, error)

Attaching to a call

// WithUse attaches middleware to a Generate call. Accepts either a middleware
// config struct or a MiddlewareFunc adapter. The chain applies outer-to-inner.
func WithUse(middleware ...Middleware) CommonGenOption

Plugin integration

// MiddlewarePlugin is implemented by plugins that provide middleware.
// Middlewares() is called during genkit.Init; returned descriptors become
// visible to the Dev UI and are reachable via the reflection API.
type MiddlewarePlugin interface {
    Middlewares(ctx context.Context) ([]*MiddlewareDesc, error)
}

Design notes

  • Config struct is the middleware. The config value the user passes to WithUse is the same type whose New method the runtime invokes. Plugin-level state rides on unexported fields and is preserved across JSON dispatch by value-copy inside the descriptor's build closure. Per-call state lives in closures captured by New.
  • No factory parameter, no prototype copy by hand. NewMiddleware captures the typed prototype in a closure that unmarshals JSON on a value copy (preserving unexported fields, sharing pointers). No BaseMiddleware embedding or field-by-field copy required.
  • Hook chains are built once per GenerateWithRequest and reused across every tool-loop iteration instead of being rebuilt per turn.
  • Local calls skip the registry entirely. WithUse(Retry{...}) invokes Retry.New(ctx) directly on the fast path; the registry is consulted only for JSON-dispatched invocations from the Dev UI or cross-runtime callers.

Other changes

  • The existing ai.ModelMiddleware / ai.WithMiddleware API is preserved and marked deprecated. New code should prefer ai.Middleware / ai.WithUse, which adds WrapGenerate and WrapTool hooks and supports dynamically injected tools via Hooks.Tools.
  • MultipartToolResponse.Content now flows through the resume path (previously dropped in handleResumedToolRequest).
  • WrapTool returns *MultipartToolResponse so metadata and content round-trip without an out-of-band capture.
  • Duplicate middleware-contributed tool names are rejected explicitly in GenerateWithRequest rather than panicking during registry registration.
  • NewToolInterruptError is exported so WrapTool hooks can interrupt tools without constructing a ToolContext.
  • Schema updates in genkit-tools and py/packages/genkit keep the middleware reference type consistent across the JS, Go, and Python runtimes.
  • Ships without any built-in implementations. See the follow-up PR for Retry, Fallback, ToolApproval, Filesystem, and Skills.

@gemini-code-assist
Copy link
Copy Markdown
Contributor

Summary of Changes

Hello @apascal07, I'm Gemini Code Assist1! I'm currently reviewing this pull request and will post my feedback shortly. In the meantime, here's a summary to help you and other reviewers quickly get up to speed!

This pull request significantly enhances the extensibility of the AI generation pipeline by introducing a comprehensive middleware system. It moves beyond simple model-level interception to offer granular control over the entire generation lifecycle, including tool usage and the iterative generation loop. This change provides a powerful mechanism for developers to inject custom logic, such as logging, metrics, or content filtering, at various critical points within the AI workflow, making the system more adaptable and observable.

Highlights

  • New Middleware System: Introduced a new, more flexible Middleware interface in Go, replacing the older ModelMiddleware. This new system provides distinct hooks for different stages of the AI generation process: Generate (for the overall tool loop), Model (for individual model API calls), and Tool (for tool executions).
  • TypeScript Type Definitions: Added new TypeScript type definitions (MiddlewareDescSchema, MiddlewareRefSchema) to genkit-tools/common/src/types/middleware.ts to support the new middleware concept, allowing middleware to be described and referenced with optional configuration.
  • Integration into Generation Options: Integrated the new middleware into GenerateActionOptions in both TypeScript and Go, allowing users to specify middleware to apply to a generation using a new Use field. The older WithMiddleware option is now deprecated.
  • Middleware Resolution and Chaining: Implemented logic in go/ai/generate.go to resolve middleware references, unmarshal their configurations, and chain their respective Model, Generate, and Tool hooks around the core generation logic. A new helper runToolWithMiddleware was added for tool hook application.
  • Plugin and Reflection API Support: Updated the Genkit Init function to register middleware provided by plugins and added a new reflection API endpoint (GET /api/values?type=middleware) to list registered middleware, enhancing discoverability and extensibility.

🧠 New Feature in Public Preview: You can now enable Memory to help Gemini Code Assist learn from your team's feedback. This makes future code reviews more consistent and personalized to your project's style. Click here to enable Memory in your admin console.

Changelog
  • genkit-tools/common/src/types/index.ts
    • Exports the newly defined middleware types.
  • genkit-tools/common/src/types/middleware.ts
    • Added new file defining TypeScript types for MiddlewareDescSchema and MiddlewareRefSchema using zod for schema validation. These describe middleware descriptors (name, description, config schema) and references (name, config).
  • genkit-tools/common/src/types/model.ts
    • Imports MiddlewareRefSchema.
    • Adds a new optional use field (an array of MiddlewareRefSchema) to GenerateActionOptionsSchema, allowing middleware to be specified for generation actions.
  • go/ai/gen.go
    • Adds a Use field (slice of *MiddlewareRef) to the GenerateActionOptions struct, mirroring the TypeScript changes and enabling middleware configuration.
  • go/ai/generate.go
    • Marks ModelMiddleware as deprecated, advising use of the new Middleware interface with WithUse.
    • Introduces logic to resolve MiddlewareRefs from GenerateActionOptions.Use into executable Middleware handlers.
    • Implements chaining for Model hooks from the new middleware, applying them before legacy ModelMiddleware.
    • Modifies handleToolRequests to accept and pass middlewareHandlers for tool hook processing.
    • Adds chaining for Generate hooks, wrapping the main generation function.
    • Introduces runToolWithMiddleware function to execute tools while applying Tool hooks from the middleware chain.
    • Updates handleResumeOption to correctly propagate the Use field.
  • go/ai/middleware.go
    • Added new file defining the Middleware interface with Name(), New(), Generate(), Model(), and Tool() hooks.
    • Defines GenerateState, ModelState, ToolState structs to pass context to middleware hooks.
    • Introduces GenerateNext, ModelNext, ToolNext type aliases for chaining functions.
    • Provides BaseMiddleware for easy implementation of middleware with default pass-through behavior.
    • Defines MiddlewareDesc for registering middleware and helper functions NewMiddleware, DefineMiddleware, LookupMiddleware.
    • Defines MiddlewareRef for serializable middleware references.
    • Introduces MiddlewarePlugin interface for plugins to provide middleware.
  • go/ai/middleware_test.go
    • Added new file containing comprehensive unit tests for the new middleware system, covering its definition, lookup, configuration, and hook invocation order for Model and Tool hooks.
  • go/ai/option.go
    • Adds a Use field (slice of Middleware) to commonGenOptions.
    • Updates applyCommonGen to handle the new Use option.
    • Deprecates WithMiddleware and introduces WithUse as the new option for applying the enhanced Middleware.
  • go/ai/prompt.go
    • Adds logic within Prompt.Execute to register dynamic middleware and build MiddlewareRefs from execOpts.Use, similar to the Generate function.
  • go/genkit/genkit.go
    • Modifies the Init function to check if a plugin implements ai.MiddlewarePlugin and, if so, registers its provided middleware descriptors.
  • go/genkit/reflection.go
    • Adds a new HTTP handler for GET /api/values to list registered values, specifically supporting filtering by type query parameter (e.g., ?type=middleware).
Activity
  • The pull request was created by apascal07.
  • The author has completed the checklist items for conventional commit title and testing.
Using Gemini Code Assist

The full guide for Gemini Code Assist can be found on our documentation page, here are some quick tips.

Invoking Gemini

You can request assistance from Gemini at any point by creating a comment using either /gemini <command> or @gemini-code-assist <command>. Below is a summary of the supported commands on the current page.

Feature Command Description
Code Review /gemini review Performs a code review for the current pull request in its current state.
Pull Request Summary /gemini summary Provides a summary of the current pull request in its current state.
Comment @gemini-code-assist Responds in comments when explicitly tagged, both in pull request comments and review comments.
Help /gemini help Displays a list of available commands.

Customization

To customize Gemini Code Assist for GitHub experience, repository maintainers can create a configuration file and/or provide a custom code review style guide (such as PEP-8 for Python) by creating and adding files to a .gemini/ folder in the base of the repository. Detailed instructions can be found here.

Limitations & Feedback

Gemini Code Assist may make mistakes. Please leave feedback on any instances where its feedback is incorrect or counter productive. You can react with 👍 and 👎 on @gemini-code-assist comments. If you're interested in giving your feedback about your experience with Gemini Code Assist for Github and other Google products, sign up here.

You can also get AI-powered code generation, chat, as well as code reviews directly in the IDE at no cost with the Gemini Code Assist IDE Extension.

Footnotes

  1. Review the Privacy Notices, Generative AI Prohibited Use Policy, Terms of Service, and learn how to configure Gemini Code Assist in GitHub here. Gemini can make mistakes, so double check it and use code with caution.

Copy link
Copy Markdown
Contributor

@gemini-code-assist gemini-code-assist bot left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Code Review

This pull request introduces a new, more powerful Middleware system for Go, which is a great enhancement. It adds hooks for Generate, Model, and Tool execution stages, deprecating the older ModelMiddleware. The implementation is solid and includes good test coverage. I've identified one bug related to data loss in tool responses and an opportunity to deduplicate some code. Overall, this is a well-executed feature.

Comment thread go/ai/generate.go Outdated
Comment thread go/ai/generate.go Outdated
@github-actions github-actions bot added the python Python label Feb 6, 2026
@apascal07 apascal07 requested a review from pavelgj February 6, 2026 16:12
@apascal07 apascal07 marked this pull request as ready for review February 6, 2026 16:12
@apascal07 apascal07 requested a review from huangjeff5 as a code owner February 6, 2026 16:12
@apascal07 apascal07 changed the title feat(go): added new Middleware feat(go/ai): added new Middleware Feb 6, 2026
@apascal07 apascal07 changed the title feat(go/ai): added new Middleware feat(go/ai): added DefineMiddleware (Middleware V2) Feb 6, 2026
@apascal07 apascal07 mentioned this pull request Feb 6, 2026
@apascal07 apascal07 linked an issue Feb 6, 2026 that may be closed by this pull request
Comment thread genkit-tools/common/src/types/index.ts
Comment thread genkit-tools/common/src/types/middleware.ts
Copy link
Copy Markdown
Contributor

@MichaelDoyle MichaelDoyle left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM from a genkit-tools perspective.

Rework the Go middleware primitives introduced in PR #4464 to collapse
configuration and behavior into a single "config struct is the middleware"
model and remove the descriptor/factory/prototype scaffolding.

- Drop the Middleware interface (Name/New/WrapGenerate/WrapModel/WrapTool/Tools)
  and the BaseMiddleware embedding helper. Introduce Hooks as a plain struct
  of optional hook func fields (WrapGenerate, WrapModel, WrapTool, Tools);
  nil hooks pass through.
- Repurpose Middleware as an interface with just Name() + New(ctx), which a
  user-facing config struct implements directly. Passing a config value to
  WithUse runs its New on the local fast path with no registry lookup, so
  pure-Go code works without plugin registration.
- NewMiddleware[M](description, prototype) captures the typed prototype in a
  closure stored on MiddlewareDesc.buildFromJSON, preserving unexported
  plugin-level state across JSON-dispatched calls via value-copy.
- MiddlewareDesc returns to being the shared schemas.config-generated type
  with the private factory added via the existing `field` directive.
- Rename MiddlewarePlugin.ListMiddleware to Middlewares to align with the
  upcoming V2 naming conventions.
- Replace Inline with MiddlewareFunc, a canonical Go adapter type that
  satisfies Middleware for ad-hoc closure-based middleware.
- Add genkit.DefineMiddleware and genkit.LookupMiddleware wrappers with
  complete godoc matching the DefineTool/LookupTool style.

Fixes carried over from the initial review:
- Preserve MultipartToolResponse.Content through the resume path in
  handleResumedToolRequest (previously dropped).
- Change WrapTool return type to *MultipartToolResponse so metadata and
  content flow through without an out-of-band capture hack.
- Reject duplicate middleware-contributed tool names explicitly in
  GenerateWithRequest instead of panicking at registry registration.
- Build the WrapGenerate, WrapModel, and WrapTool hook chains once per
  GenerateWithRequest rather than rebuilding them on every tool-loop turn.
- Export NewToolInterruptError so WrapTool hooks can interrupt tools
  without constructing a ToolContext.

Tests rewritten against the new shape and expanded to cover: plugin-state
value-copy, call-level state isolation, MiddlewareFunc adapter, nil hooks,
stream chunk accumulation, tool contribution, duplicate-tool rejection,
factory error propagation, WrapTool interrupts, per-iteration WrapGenerate,
and metadata round-trip through WrapTool. All green under -race.
@apascal07
Copy link
Copy Markdown
Collaborator Author

apascal07 commented Apr 17, 2026

@pavelgj Minor but pretty great improvements to the middleware abstraction in case you want to review it again. The PR description covers it all.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

RFC: Middleware V2

4 participants